Distributing and processing streams over one or more networks for on-the-fly schema evolution

ABSTRACT

In an embodiment, a method comprises receiving a first record comprising one or more first fields from a first source computer; in response to determining that the first record is a first type and a first node is associated with the first type, sending the first record to the first node to be processed; receiving a second record comprising the one or more first fields and one or more second fields from a second source computer; in response to determining that the second record is a second type and a second node is associated with the second type, sending the second record to the second node; in response to determining that the second type is a first subtype of the first type, sending the second record to the first node to be processed, without suspending the first node.

BENEFIT CLAIM

This application claims the benefit under 35 U.S.C. §120 as acontinuation of application Ser. No. 14/341,676, filed Jul. 25, 2014,the entire contents of which are hereby incorporated by reference forall purposes as if fully set forth herein. The applicants hereby rescindany disclaimer of claim scope in the parent applications or theprosecution history thereof and advise the USPTO that the claims in thisapplication may be broader than any claim in the parent applications.

FIELD OF THE DISCLOSURE

The present disclosure relates to computers and computer-implementedtechniques useful in management of computer networks including networkmanagement databases relating to analytics. The disclosure relates morespecifically to techniques for processing continuous queries of networkmanagement data without suspending and reconfiguring currently executingqueries.

BACKGROUND

The approaches described in this section could be pursued, but are notnecessarily approaches that have been previously conceived or pursued.Therefore, unless otherwise indicated herein, the approaches describedin this section are not prior art to the claims in this application andare not admitted to be prior art by inclusion in this section.

Data communication networks may comprise one or more computing nodes.Each node may receive and process unbounded, streaming data according toa particular schema. A schema may define the content, and/or the formatof the content, in a data stream. Each node may send results to one ormore downstream nodes, persistent storages, and/or some other receivingdevice, according to a schema.

In a modern data-driven system, the schema or format of a data streammay change at will. For example, a schema for a network data stream mayinclude frequent status reports for each mission critical device on thenetwork, which includes the device's Internet Protocol address and atime stamp and may change to include the power consumption, CPU usage,and available memory on each device.

A change in a data stream's schema may cause may cause a node to stop.However, some scenarios require that a node remain processing, in whichcases the node is not allowed to be suspended or reconfigured to takeadvantage of a schema change.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates a plurality of records that are one or more types inan example embodiment.

FIG. 2 illustrates a system for processing different types and subtypesof streaming input data from a plurality of sources without suspendingone or more nodes in an example embodiment.

FIG. 3 illustrates a process for processing a plurality of record typesin one example embodiment.

FIG. 4 illustrates a computer system upon which an embodiment may beimplemented.

While each of the drawing figures illustrates a particular embodimentfor purposes of illustrating a clear example, other embodiments mayomit, add to, reorder, and/or modify any of the elements shown in thedrawing figures. For purposes of illustrating clear examples, one ormore figures may be described with reference to one or more otherfigures, but using the particular arrangement illustrated in the one ormore other figures is not required in other embodiments.

DETAILED DESCRIPTION

Techniques for distributing and processing independent data streams overone or more networks are described. In the following description, forthe purposes of explanation, numerous specific details are set forth inorder to provide a thorough understanding of the present disclosure. Itwill be apparent, however, to one skilled in the art that the presentdisclosure may be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to avoid unnecessarily obscuring the present disclosure.

Embodiments are described herein according to the following outline:

-   -   1.0 Overview    -   2.0 Structural and Functional Overview        -   2.1 Subtypes, Super-Types, Subsets, and Supersets            -   2.1.1 Subtypes, Super-Types, Subsets, and Supersets            -   2.1.2 Casting Instances of Subtypes to Super-Types        -   2.2 Example System Topology            -   2.2.1 Routing Module            -   2.2.2 Nodes            -   2.2.3 Sources            -   2.2.4 Record Storage    -   3.0 Process Overview        -   3.1 Receiving Continuous Queries        -   3.2 Generating, Instantiating, and/or Initializing Nodes            based on the Continuous Queries        -   3.3 Determining Subtypes        -   3.4 Processing Records        -   3.5 Casting Records without Suspending Nodes        -   3.6 Processing Cast Records without Suspending Nodes        -   3.7 Processing New, Later-Defined Types of Records without            Suspending Nodes    -   4.0 Implementation Mechanisms—Hardware Overview    -   5.0 Extensions and Alternatives

1.0 Overview

In an embodiment, a computer system comprises a first node computercomprising a first processor and a first memory configured to processone or more first type of records based on a first operation; a secondnode computer comprising a second processor and a second memoryconfigured to process one or more second type of records based on asecond operation; a routing computer comprising a processor, a memory,and a routing module configured to: receive a first record comprisingone or more first fields from a first source computer; in response todetermining that the first record is a first type and the first nodecomputer is associated with the first type, send the first record to thefirst node computer to be processed; receive a second record comprisingthe one or more first fields and one or more second fields from a secondsource computer; in response to determining that the second record is asecond type and the second node computer is associated with the secondtype, sending the second record to the second node computer to beprocessed; in response to determining that the second type is a firstsubtype of the first type, sending the second record to the first nodecomputer to be processed, without suspending and reconfiguring the firstnode computer.

In an embodiment, a method comprises receiving a first record comprisingone or more first fields from a first source computer; in response todetermining that the first record is a first type and a first node isassociated with the first type, sending the first record to the firstnode to be processed; receiving a second record comprising the one ormore first fields and one or more second fields from a second sourcecomputer; in response to determining that the second record is a secondtype and a second node is associated with the second type, sending thesecond record to the second node; in response to determining that thesecond type is a first subtype of the first type, sending the secondrecord to the first node to be processed, without suspending the firstnode.

In an embodiment, the method comprises in response to determining thatthe first record is the first type and the first node is associated withthe first type, sending a first signature with the first record to thefirst node; in response to determining that the second record is thesecond type and the second node is associated with the second type,sending a second signature with the second record to the first node; inresponse to determining that the second type is the first subtype of thefirst type, sending the first signature with the second record to thefirst node.

In an embodiment, the method comprises organizing the one or more firstfields in the first record according to one or more first indices;organizing the one or more first fields in the second record accordingto the one or more first indices; organizing the one or more secondfields in the second record according to one or more second indices,wherein each index in the one or more first indices references a memoryaddress that is before the memory address referenced by each index inthe one or more second indices.

In an embodiment, the method comprises receiving, from a third sourcecomputer, a third record comprising the one or more first fields, theone or more second fields, and one or more third fields; in response todetermining that the third record is a third type and a third node isassociated with the third type, sending the third record to the thirdnode to be processed; in response to determining that the third type isa second subtype of the second type, sending the third record to thesecond node to be processed without suspending and reconfiguring thesecond node; in response to determining that the second type is thefirst subtype of the first type, sending the third record to the firstnode to be processed without suspending and reconfiguring the firstnode.

2.0 Structural and Functional Overview

Data communication networks may comprise one or more computing nodes.Each node may receive and process unbounded, streaming data according toa particular schema. A schema may define the content, and/or the formatof the content, in a data stream. Each node may send results to one ormore downstream nodes, persistent storages, and/or some other receivingdevice, according to a schema.

Suspending a node may cause many problems. Suspending may includehalting a node and/or otherwise causing a node to stop processing one ormore input data streams. For example, a node may process time-criticaldata, such as networking, power, or rocket telemetry data. If the nodeis suspended, a downstream node or device may be unable to perform itsfunctions properly or may derive misleading data from the lack of datareceived for the suspended node. For example, the downstream node ordevice may determine that a network is down or a rocket has lostcontact, however the data not processed by the suspended node mayindicate that a network is excellent connectivity with one or more othernetworks or that a rocket is performing within mission parameters.

Suspending a node may cause massive reserve resources dedicated to thenode or data loss. For example, a node may process a data stream thatcontinuously delivers a large amount of data. If the node is suspended,then a large amount of data may begin to build up, which may need to bestored in a large persistent data store. Even data may still be losteven if unprocessed data is stored in the large persistent data store.For example, if the node is offline for too long, then the data storefill to capacity. Thus, either new data may not be stored, or the older,unprocessed data may be deleted to make room for the newly receiveddata. Furthermore, if the bandwidth of the data received is greater thanthe bandwidth of the persistent storage, then some data may be lost oroverwritten while waiting in a volatile buffer waiting to be written topersistent storage.

Suspending a node may cause one or more data sources to fail. Forexample, if a data source receives an exception error because areceiving node is suspended, then the data source may exit. Thus, evenwith the suspended node resumes processing the data source may haveterminated.

In a modern data-driven system, the schema or format of a data streammay change at will. For example, a schema for a network data stream mayinclude frequent status reports for each mission critical device on thenetwork, which includes the device's Internet Protocol address and atime stamp and may change to include the power consumption, CPU usage,and available memory on each device.

In response to schema changes, a node may be suspended, reconfigured,and resumed. Continuing with the previous example, if the data consumeradministrating the node wishes to process and/or use the new data in thedata stream (the power consumption, CPU usage, and available memory oneach device), then the administrator may suspend the node, reconfigurethe node to make use of the new data, and resume the node. The data sentwhile the node is offline may be lost or, if possible, stored in apersistent storage when the node comes back online.

After a node resumes processing, the node may not immediately beginprocessing the most recent data. Continuing with the previous example,when the reconfigured node resumes, the node may process the data storedin persistent storage first, meanwhile newly received data may continueto be stored in persistent storage or lost. For example, if thepersistent storage's bandwidth is insufficient to support the readsrequested by the node and the writes from the data stream, then the datain the data stream may be lost while the data from persistent storage isprocessed by the node.

A change in a data stream's schema may cause a node to be suspended,which in turn may cause many problems, as discussed above, and may beprone to error. Furthermore, some scenarios require that a node remainprocessing, in which cases the node is not allowed to be suspended orreconfigured to take advantage of a schema change. Thus, gracefully andefficiently supporting schema changes without downtime is extremelyvaluable.

One or more systems and methods are discussed herein to process inputdata streams that conform to changing schemas without suspending and/orreconfiguring running nodes. Suspending a node may mean stopping a nodefrom processing and/or executing one or more continuous queries that arebased on a first type of record. A continuous query may comprise one ormore continuous queries, sub-queries, operations defined in a continuousquery, and/or sub-operations defined in a continuous query.Reconfiguring a node may mean suspending a node and/or causing the nodeto execute and/or process one or more new continuous queries based on asecond, different type of record that is a subtype of the first type ofrecord. Additionally or alternatively, reconfiguring a node may meancausing a node to execute and/or process one or more new continuousqueries that perform a super set of the functionality of the one or morefirst continuous queries.

Data streams may comprise one or more records sent over time. A recordin a data stream may be organized according to a schema. Records thatare organized according to a first schema may be referred to as a “firsttype” or a “first type of record”. Records that are organized accordingto a second schema may be referred to as a “second type” or “second typeof record”.

One or more systems and methods are discussed herein for a first node,configured to process a first type of record, to also process a secondtype of record without suspending and/or reconfiguring the first node.For example, a record that is a second type of record may be cast as thefirst type of record and processed by the first node without suspendingand/or reconfiguring the first node. Accordingly, one or more systemsand methods are discussed herein to cast a second type of record as afirst type of record, and process the second type of record as if therecord was originally received as a first type of record.

2.1 Records and Types

A record may comprise one or more fields. A record may also comprise anexpress signature. A “bare” record does not include an expresssignature. A bare record may, but need not, include an impliedsignature. A signature may include one or more values and/or datastructures that identify the record a particular type of record. One ormore of the systems and/or methods discussed herein may append and/orprepend a signature to a record, such that the record is no longer abare record.

FIG. 1 illustrates a plurality of records that are one or more types inan example embodiment. In FIG. 1, record 110 is an example first type ofrecord and comprises signature 112 and fields 114A. Signature 112 mayidentify record 110 as a first type of record. Fields 114A may compriseone or more values, key-value pairs, lists of values and/or key-valuepairs, hierarchical data structures, pointers, and/or any other datastructures.

Record 120 is an example second type of record and comprises signature122, fields 114B, and fields 124A. Signature 122 may identify record 120as a second type of record. Fields 114B in record 120 has the sameformat and/or data structure(s) as fields 114A in record 110, but thevalues may be different. Fields 124A may comprise one or more differentand/or additional values, key-value pairs, lists of values and/orkey-value pairs, hierarchical data structures, pointers, and/or anyother data structures that are not included in fields 114B.

In the example records illustrated in FIG. 1, each record includes an“express” signature comprising value and/or data structure. For example,record 110 includes signature 112, and record 120 includes signature122. However, in an embodiment, a signature may be implied by one ormore names, values, and/or data structures of one or more fields in therecord. For example, any record that includes a field named “address”may be determined to be an instance of a first type, and any record thatincludes a field named “distance” may be determined to be an instance ofa second type. A record that includes both a field name “address” and afield name “distance” may be determined to be an instance of both thefirst type and the second type.

2.1.1 Subtypes, Super-Types, Subsets, and Supersets

A second type of record may be referred to herein as a “subtype” of afirst type of record if the fields in the second type of record comprisea superset of the required fields in the first type of record. The firsttype may be referred to herein as a “super-type” and/or a “subset” ofthe second type. For purposes of illustrating a clear example, assumethat the one or more fields in fields 114A are required for a first typeof record, and the one or more fields in fields 114B and fields 124A arerequired for any instance of a second type of record. In the currentexample, a record that is an instance of the second type also includesthe required fields to be an instance of the first type. Thus, thesecond type is a “subtype” or “superset” of the first type; and, thefirst type is a “super-type” or “subset” of the second type.

A field may be a required field if it is used by one or more nodes thatprocess the first type of records. Additionally or alternatively, afield may be a required field if the field is designated as required ina schema. For example, a first schema, which defines the first type ofrecord, may include a list of the names of the fields that are required.Additionally or alternatively, the first schema may identify one or morefields in a first data structure as required and one or more fields in asecond data structure as optional.

A record may have more than one subtype and/or super-type. For example,record 130 is an example of a third type of record. Record 130 comprisessignature 132, fields 114C, fields 124B, and fields 134. Signature 132may identify record 130 as a third type of record and/or as a recordthat may be processed by one or more third nodes. Fields 114C in record130 has the same format and/or data structure(s) as fields 114A inrecord 110, but the values may be different. Fields 124B in record 130has the same format and/or data structure(s) as fields 124A in record120, but the values may be different. In the embodiment illustrated inFIG. 1, the third type of record is a subtype of the first type ofrecord, because the third type of record has the one or more requiredfields in the first type (fields 114A in record 110 and fields 114C inrecord 130). The third type of record is a subtype of the second type ofrecord, because the third type of record has the one or more requiredfields in the second type (fields 124A in record 120 and fields 124B inrecord 130).

The signature and fields of each record in FIG. 1 are illustrated as ifcontiguous in memory. However, the signature and/or fields of eachrecord need not be contiguous. For example, signature 122, fields 114B,and fields 124A in record 120 may, but need not, be stored contiguouslyin memory. Furthermore, each field in fields 114A and/or fields 114Bmay, but need not, be stored contiguously in memory.

2.1.2 Casting Instances Of Subtypes To Super-Types

Records that are a subtype of another type may be cast as a super-typeby changing the signature, fields, and/or data structures in therecords. For example, record 121 is a copy of record 120, which is asecond type of record, cast as a super-type: the first type. Record 121comprises signature 112, fields 114B, and fields 124A. Just as in record110, signature 112 in record 121 may include one or more values and/ordata structures that identify record 121 as a first type and/or as arecord to be processed by one or more first nodes associated withsignature 112 and/or the first type. However, the one or more firstnodes may disregard fields 124A in record 121.

One or more fields that are not in a super-type may be stored highermemory address space. For example, in FIG. 1, memory addresses grow fromleft to right. Thus, in record 120, a first index that references fields114B is less than a second index that references fields 124A. A firstnode that processes the first type of record may use the same index toreference fields 114A in record 110 and fields 114B in record 121,because fields 124A are assigned a higher memory address. Thus, thefirst node may process record 121 as if record 121 was originally afirst type of record.

Common fields between a subtype and a super-type may be stored in thesame order. For example, fields 114C and fields 124B in record 130 arein the same order as fields 114B and fields 124A in record 120. A nodethat processes the second type of records may use the same indices toreference fields 114B and fields 124A in record 120 as fields 114C andfields 124B in record 130. Furthermore, a node that processes the firsttype of records may use the same index to reference fields 114A inrecord 110, fields 114B in record 120, fields 114B in record 121, andfields 114C in record 130.

2.2 Example Multi-Node System Topology

FIG. 2 illustrates a system for processing different types and subtypesof streaming input data from a plurality of sources without suspendingone or more nodes in an example embodiment. In FIG. 2, system 200includes node computer cluster 210, routing computer 220, record storagedevice 224, source computer 240, source computer 241, source computer250, and client computer 290 distributed across a plurality ofinterconnected networks. Node cluster 210 includes source node computer242, query node computer 244, source node computer 252, query nodecomputer 254, and node computer 212, distributed across one or moreinterconnected networks and may perform one or more of the methoddiscussed herein.

A computer may be one or more software modules, computers, computercomponents, computing devices, databases, data stores, routers,switches, and/or other elements of internetworking infrastructure.Unless expressly stated as a “single” computer, a computer may be one ormore computers.

More than one node may be executed on a single computer. For example,node computer 212 includes query node 246 and query node 256, which areexecuted on node computer 212, based on operating system 216, and atleast partially reside in memory device 214.

While one or more of the components listed above may be illustrated asif running on a separate, remote computer from each other, one or moreof the components listed above may be part of and/or executed on thesame computer. For example, routing module 222, record storage device224, source computer 250, source node computer 252, query node computer254, and/or any of the devices and/or modules included therein, may beexecuted on the same, single computer, local area network, and/or widearea network.

2.2.1 Routing Module

Routing computer 220 includes routing module 222. Routing module 222 mayreceive one or more requests and/or queries from one or more clientcomputers, generate one or more nodes, receive one or more records fromone or more sources, nodes, modules, and/or computers, and/or send oneor more records to the one or more other sources, nodes, modules, and/orcomputers. Additionally or alternatively, routing module 222 may storerecords in record storage device 224. Additionally or alternatively,routing module 222 may send records and/or results to client computer290 and/or any other modules, nodes, and/or computers.

A module may be software executed on a computer and/or a hardware thatincludes logic, and/or is coupled to, a computer. A module may includeinstructions and/or logic, which when executed cause performance of oneor more of the methods discussed herein.

A routing module may receive and/or execute one or more continuousqueries. A continuous query is a query for data, including a queryagainst a database, which defines one or more functions to perform onstreaming input data identified as one or more particular types. Acontinuous query may be defined by a query language, such as astructured query language or a continuous query language. A continuousquery can be issued either as, or as part of, a database query or a datadefinition language command. The one or more functions defined by acontinuous query may be database specific functions, networkingfunctions, analytic functions, multimedia functions, and/or any otherstreaming data functions. For example, a continuous query may specifysummarizing, aggregating, or filtering network analytics data in anetwork management application. A continuous query may define one ormore data stream sources, schemas, types, subtypes, and/or super-types.A continuous query may define one or more operations assigned to one ormore nodes. The continuous query may instruct the one or more nodes towait for data to arrive, process the data, and then output derived,processed data. The features and processes discussed herein may be usedfor various purposes, such as streaming analytic data, debugging anddiagnostics, recovery from failures, as well as in storing andprocessing previously received streaming data.

Routing module 222 may generate one or more types of nodes to executeone or more types of records based on one or more continuous queries. Afirst node may be a “first type of node” if the node processes a firsttype of record. Similarly, a second node may be a “second type of node”if the node processes a second type of record. In response to receivinga continuous query defining a first type of record received from sourcecomputer 240, routing module 222 may generate, instantiate, and/orinitialize source node computer 242, query node computer 244, and querynode 246 on node computer 212.

Routing module 222 may orchestrate which records are sent to which nodesand in which order. For example, routing module 222 may associate afirst type of record, defined in a first continuous query, with thefollowing nodes in the following order: source node computer 242, querynode computer 244, and query node 246. When a first type of record isreceived, routing module 222 may route the record to source nodecomputer 242. Based, at least in part, on routing module 222, sourcenode computer 242 may send the record, and/or data derived from therecord, to query node computer 244. Based, at least in part, on routingmodule 222, query node computer 244 may send the record, and/or dataderived from the record, to query node 246. Based, at least in part on,routing module 222, query node 246 may send the record, and/or dataderived from the record, to client computer 290, record storage device224, and/or any other computer and/or module defined by the continuousquery. Accordingly, routing module 222 may comprise one or more tables,databases, and/or data structures for storing associations between oneor more continuous queries, types of records, signatures, and/or nodes.

Routing module 222 may manage casting records from a subtype to asuper-type and processing the “up-casted” record with the nodesassociated with the super-type. For example, routing module 222 mayreceive a continuous query that defines a second type, and indicatesthat the second type is a subtype of a first type. In response, routingmodule 222 may generate, instantiate, and/or initialize the followingnodes to process the second type of records in the following order:source node computer 252, query node computer 254, and query node 256.In response to receiving a second type of record, routing module 222 maycast the second type of record as a first type of record, and/or sendthe up-casted record to the following nodes associated with the firsttype, in the following order: source node computer 242, query nodecomputer 244, and query node 246. Additionally or alternatively, basedon routing module 222, a node may up-cast a record and/or send theup-casted record to one or more nodes in a particular order.

Conventionally, when a record with an updated schema is received, eachnode previously processing the records may be suspended, reconfigured toprocess the new data, and resumed. As discussed herein, suspending oneor more nodes can cause many problems. In contrast, routing module 222is unconventional by instantiating new nodes to process the new data ina subtype or record, casting a subtype of record to a super-type ofrecord, and routing the up-casted record to already-running nodes toprocess the data common between the subtype of record and the super-typeof record, without suspending the already-running nodes. Thus, for theseand many other reasons discussed herein, the unconventional methodsand/or systems discussed herein, such as the methods and systemsrelating to routing module 222, improve the functioning of thecomputer(s) and/or computer system(s).

2.2.2 Nodes

A node may be a module, computer, computer component, computing device,database, data store, router, switch, and/or other element ofinternetworking infrastructure. A node may be defined by circuitry, oneor more software instructions, a query, and/or a continuous query. Anode may include logic and/or instructions, which when executed performone or more of the methods discussed herein.

Each node may be configured to perform one or more different tasks,processes, and/or operations than another node. For example, a sourcenode, such as source node computer 242, may receive, format, enqueue,dequeue, and/or send records to one or more nodes, routing modules,and/or storage devices. Source node computer 242 may be specialized toprocess records that are a first type. Source node computer 252 mayperform similar functions as source node computer 242, but may bespecialized to process records that are a second type.

Query node computer 244 may be configured to eliminate one or more firsttype records that do not satisfy one or more criteria, and send therecords that do satisfy the one or more criteria to query node 246.Query node 246 may perform one or more operations on the first type ofrecords received from query node computer 244, such as a sum, sort,and/or one or more other operations.

Query node computer 254 may perform one or more different operations ona second type of record, and/or data derived from a second type ofrecord, than query node computer 244 processing an up-casted version ofthe same record, and/or data derived from the up-casted version of samerecord. Query node 256 may perform one or more different operations on asecond type of record, and/or data derived from a second type of record,than query node 246 processing an up-casted version of the same record,and/or data derived from the up-casted version of same record.

A node that receives one or more records from one or more other nodesmay be referred to as a “query node”. For example, source node computer242 may be a source node and query node computer 244 and query node 246may each be a query node. However, in an embodiment, a source node mayperform one or more of the operations that a query node may perform,and/or vice versa. For example, source node computer 242 may receive arecord from source node computer 252. Also for example, a source nodecomputer 242 may receive and process a first type of record and need notsend the record and/or data derived from the record to a downstreamnode.

A node may return results to routing module 222, client computer 290,and/or another module and/or computer. For example, query node 246 mayreceive one or more records indicating the CPU usage for one or morecomputers. Query node 246 may determine whether the CPU usage for eachcomputer is greater than a particular threshold. If so, query node 246may send a message to routing module 222 and/or client computer 290indicating which computer(s) have a CPU usage greater than theparticular threshold.

A node may add, remove, and/or change data in a record. For example,query node 246 may append and/or prepend data to each record indicatingwhether the record included data that satisfied one or more criteria.Query node 246 may send each modified record to routing module 222,record storage device 224, client computer 290, and/or another nodeand/or computer. Also for example, source node computer 242 may receivea record, and/or a copy of a record, from source node computer 252.Source node computer 252 may have already prepended a signatureassociated with the second type to the record. Source node computer 242may determine that the signature is associated with the second type,determine that the second type is associated with the first type as asubtype based on routing module 222, and/or replace the signature with asignature associated the first type.

A node may send a record, modified record, and/or data derived from arecord to another node and/or computer. For example, source nodecomputer 242 may send the record to query node computer 244 and/orrouting module 222 for further processing. Also for example, routingmodule 222 may receive the record from source node computer 252. Routingmodule 222 may modify the record and/or send the record to source nodecomputer 242.

In FIG. 2 a node may send and/or stream records to another node and/orcomputer directly. For example, source node computer 242 may behardwired and/or hardcoded to send processed records to one or morenodes, such as query node computer 244. Also for example, source nodecomputer 242 may query routing module 222 for an address and/oridentifier of the node that source node computer 242 should send aprocessed record to, based, at least in part, on a signature of theprocessed records and/or any other factor discussed herein. In thisexample, routing module 222 may comprise a relational database thatassociates the signature of the processed record with the address ofquery node computer 244. Thus, routing module 22 may return the addressof query node computer 244 to source node 242, and source node 242 maysend the process record query node computer 244. Additionally oralternatively, a node may send and/or stream data to routing module 222and/or record storage device 224. Routing module 222 and/or recordstorage device 224 may push records to one or more nodes. Additionallyor alternatively, a node may poll for data from routing module 222and/or record storage device 224. For example, query node computer 244may query routing module 222 and/or record storage device 224 forrecords that source node computer 242 has processed and/or stored inrecord storage device 224.

2.2.3 Sources

A source computer may be one or more computers streaming one or moreparticular types of records to one or more source nodes. For example,source computer 240 may be a first generation router on a first network,source computer 241 may be a first generation router on a secondnetwork, and source computer 250 may be a second generation router on athird network. Source computer 240 and source computer 241 may each senda plurality of records that are a first type of record to source nodecomputer 242. Source computer 250 may send a plurality of records thatare a second type of record to source node computer 252.

In FIG. 1 each of the source computers stream records directly to asource node. Additionally or alternatively, source computers may streamdata to routing module 222 and/or record storage device 224. Routingmodule 222 and/or record storage device 224 may push records to eachsource node. Additionally or alternatively, each source node may pollfor data from routing module 222 and/or record storage device 224.

2.2.4 Record Storage

Record storage device 224 may receive, store, and/or send one or morerecords. For example, source node computer 242 may receive a pluralityof records from source computer 240. Source node computer 242 mayrequest and/or process the records. Source node computer 242 may sendthe record, and/or data derived from the record, to record storagedevice 224 to be stored. As query node computer 244 is ready to processthe records, query node computer 244 may request and/or receive therecords over time from record storage device 224. Additionally oralternatively, routing module 222 may request one or more records inrecord storage device 224 and send the one or more records to a nodeand/or computer. Additionally or alternatively, routing module 222 maystore one or more records and/or data sent to routing module 222 fromone or more nodes and/or computers.

3.0 Process Overview

The systems, methods, and/or data structures discussed herein may beused to receive and process records with dynamic and/or evolving schemaswithout stopping and/or reconfiguring nodes that are already executing.FIG. 3 illustrates a process for processing a plurality of record typesin one example embodiment. For purposes of illustrating a clear example,assume that a network administrator is using client computer 290 toanalyze and monitor network connectivity between three networks. Thefirst network is communicatively coupled with one or more networksthrough source computer 240, which is a first generation router; thesecond network is communicatively coupled with one or more networksthrough source computer 241, which is also first generation router; andthe third network is communicatively coupled with one or more networksthrough source computer 250, which is a second generation router.Records sent from a first generation router are a first type of record,records sent from a second generation router are a second type ofrecord, and the second type is a subtype of the first type. Accordingly,each record that is a second type may include a “BGP Next Hop” field,whereas a first type of record need not.

3.1 Receiving Continuous Queries

In step 305, a routing module associates a first signature with a firsttype and second signature with a second type. For example, the networkadministrator, through client computer 290, may send two continuousqueries to routing module 222. The first continuous query may define afirst signature for a first type of record to be the value signature112, and the second continuous query may define a second signature for asecond type of record to be the value signature 122. Accordingly,routing module 222 may update and/or store data in routing module 222,routing computer 220, a database, and/or any other computer coupled withrouting module 222, which associates signature 112 with the first typeof record and signature 122 with the second type of record.

3.2 Generating, Instantiating, and/or Initializing Nodes Based On TheContinuous Queries

In step 310, the routing module associates the first type with the firstnode and a second type with a second node. For example, the firstcontinuous query may define one or more operations to be performed oneach first type of record. Routing module 222 may analyze the one ormore operations and generate, instantiate, and/or initialize source nodecomputer 242 to receive one or more first type of records, query nodecomputer 244 to perform a first operation defined in the firstcontinuous query, and query node 246 to perform a sub-operation definedin the first continuous query. Similarly, the second continuous querymay define one or more operations to be performed on each second type ofrecord. Routing module 222 may analyze the one or more operations andgenerate, instantiate, and/or initialize source node computer 252 toreceive one or more second type of records, query node computer 254 toperform a first operation defined in the second continuous query, andquery node 256 to perform a sub-operation defined in the secondcontinuous query.

3.3 Determining Subtypes

In step 315, the routing module associates the second type of record asa subtype of the first type of record. For example, the secondcontinuous query may include data that expressly indicates that recordsof the second type are a subtype of the first type. In response, routingmodule 222 may associate the second signature (signature 122) with thefirst signature (signature 112), which indicates that records with thesecond signature are instances of a subtype of the first type.Additionally or alternatively, routing module 222 may analyze thefield(s) expected to be read, and/or written to, by nodes initializedfor the second continuous query, and may determine that those fields area superset of the fields expected to be read, and/or written to, bynodes initialized for the first continuous query. Accordingly, routingmodule 222 may update and/or store data in routing module 222, routingcomputer 220, a database, and/or any other computer coupled with routingmodule 222, which indicates that the second type is a subtype of thefirst type.

3.4 Processing Records

In step 320, the first node and the second node receive a first recordand a second record, respectively. For example, source computer 240 maysend a first record, which may be a bare record and/or first type, tosource node computer 242. Source computer 250 may send a second record,which may be a bare record and/or second type, to source node computer252.

Additionally or alternatively, source computer 240 and source computer250 may send the first record and the second record to routing module222. Routing module 222 may send the first record to source nodecomputer 242 and the second record to source node computer 252.

Additionally or alternatively, routing module 222 may store the recordsin record storage device 224. Source node computer 242 may requestrecords associated with the first type from routing module 222. Inresponse, routing module 222 may send the first record to source nodecomputer 242. Similarly, source node computer 252 may request recordsassociated with the second type from routing module 222. In response,routing module 222 may send the second record to source node computer252. Source node computer 242 and source node computer 252 may poll forrecords based on the one or more continuous queries, having finishedprocessing one or more previously received records, an amount of timethat has elapsed, available central processing unit cycles, availablememory, and/or any other factor discussed herein and/or related to oneor more computers executing source node computer 242, source nodecomputer 252, routing module 222, and/or record storage device 224.

In step 325, the first source node determines that the first record is afirst type and the second source node determines that the second recordis a second type. For example, source node computer 242 may determinethat the first record is a first type based on the source, format,schema, and/or any other express and/or implied attribute(s) of thefirst record. Source node computer 252 may determine that the secondrecord is a second type based on the source, format, schema, and/or anyother express and/or implied attribute(s) of the second record. Anexpress attribute may be any data and/or data structures included in therecord, such as one or more field names. An implied attribute may be anydata inferred from the record, such as the address of the sourcecomputer the record was received from, the make and/or model of thesource computer the record was send from, and/or the time at which therecord was received.

Additionally or alternatively, routing module 222 may determine that thefirst record is the first type and/or the second record is the secondtype using one or more of the factors discussed herein. In anembodiment, routing module 222 may determine the type of each recordwhen the record is received.

In step 330, the first source node associates the first signature withthe first record and the second signature with the second record. Forpurposes of illustrating a clear example, assume that the first recordis record 110 and the second record is record 120; however, record 110need not include signature 112 and record 120 need not include signature122. After receiving record 110, source node computer 242 may prependthe first signature (signature 112) to the record 110, such that thesignature 112 is the first datum in record 110. After receiving record120, source node computer 252 may prepend the second signature(signature 122) to record 120, such that signature 122 is the firstdatum in record 120.

Additionally or alternatively, routing module 222 may associate thefirst signature with the first record and the second signature with thesecond record by prepending the first signature to the first record andthe second signature to the second record. Routing module 222 store eachrecord in record storage device 224. Each record stored by routingmodule 222 may include an express signature.

In step 335, the first source node sends the first record to a firstquery node and the second source node sends the second record to asecond query node. For example, source node computer 242 may send record110 to query node computer 244, and source node computer 252 may sendrecord 120 to query node computer 254. Query node computer 244 mayprocess record 110 and query node computer 254 may process record 120.

3.5 Casting Records Without Suspending Nodes

In step 340, the source nodes may determine whether the record(s) theyeach received belongs to a subtype of another type. If so, then controlmay pass to step 345. Otherwise, control may pass to step 390. Forexample, source node computer 242 may determine that record 110 is notan instance of a subtype based on one or more of the following:signature 112 is not associated with another signature as a subtype;record 110 is not a superset of another type of record; and/or any otherfactor(s) discussed herein. Accordingly, source node computer 242 mayproceed to step 390.

Additional or alternatively, source node computer 242 may send record110 to routing module 222. Routing module 222 may determine whetherrecord 110 is an instance of a subtype of another type of record basedon one or more of the methods discussed herein. If so, then routingmodule 222 may pass to step 345. Otherwise, routing module 222 may passto step 390.

As another example of step 340, source node computer 252 may determinethat record 120 is an instance of a super-type (the first type) based onone or more of the factors discussed herein, and proceed to step 345.Additionally or alternatively, source node computer 252 may send record120 to routing module 222. Routing module 222 may determine record 120is an instance of the first type based on one or more of the factorsdiscussed herein, and proceed to step 345.

In step 345, the first source node and/or the routing module mayassociate the record with the super-type's signature. For example,source node computer 242 may receive a copy of the second record (record121) from source node computer 252 and/or routing module 222. Record 121may, but need not, be a bare record. Thus, record 121 may, but need not,include signature 122. Source node computer 242 and/or routing module222 may prepend signature 112, and/or remove the signature 122, fromrecord 121 as illustrated in FIG. 1.

3.6 Processing Cast Records Without Suspending Nodes

In step 350, the first source node may send the record with thesuper-type's signature to the node associated with the super-type'ssignature. For example, source node computer 242 and/or routing module222 may send record 121 to the first query node associated with thefirst type: query node computer 244. Control may then return to step340. If, for example, record 120 and/or record 121 is a subtype of athird type, then the process may be repeated. Otherwise, control mayproceed to step 390. In the current example, record 121 is not a subtypeof an additional type, thus source node computer 242 and/or routingmodule 222 may proceed to step 390.

In step 390, control may terminate, wait for new records, and/or pollfor new records. For example, source node computer 242 and/or routingmodule 222 may terminate, wait for one or more new records, and/or queryand/or poll for one or more new records.

3.7 Processing New, Later-Defined Types of Records Without SuspendingNodes

In the foregoing example, the records that were a second type andreceived from the second generation router (source computer 250), wereprocessed by both the plurality of nodes configured to process thesecond type of records (nodes 252-256) and the plurality of nodesconfigured to process the first type of records (nodes 242-246). Thus,source node computer 252, query node computer 254, and query node 256could process the additional data in the second type of records (the“BGP Next Hop” field), and source node computer 242, query node computer244, and query node 246 could still process each second type of recordas if a first type of record without being suspended or reconfigured.

In the foregoing example, the first and the second continuous query werereceived before any record of any type was received. However, the secondcontinuous query and/or the second type of records could be receivedafter the first continuous query and/or the first type of records werereceived and/or began being processed by any node. Similarly, nodes252-256 could have been generated, instantiated, and/or initializedafter nodes 242-246 began processing the first type of records.

If a third generation router began streaming a third type of record tothe system illustrated in FIG. 2, and the third type of record was asubtype of the second type of record, then routing module 222 couldgenerate, instantiate, and/or initialize one or more new nodes toprocess the additional data included in the third type of records.Routing module 222 may also cause each third type of record to be castto a second type of record and processed by nodes 252-256, using one ormore of the methods discussed above, without suspending and/orreconfiguring nodes 252-256. Additionally or alternatively, routingmodule 222 may also cause each third type of record to be cast to afirst type of record and processed by nodes 242-246, using one or moreof the methods discussed above, without suspending and/or reconfiguringnodes 242-246.

Using one or more of the methods discussed above, if a first type ofrecord in a data stream was being processed according to a firstcontinuous query by a first node, and the schema of the records in thedata stream is updated to include a new field, then the routing modulemay send the records (which are now a second type, and a subtype of thefirst type) in the data stream to a second, new node. The second nodemay begin processing the data in the new field in each second type ofrecord according to a second, new continuous query. The routing modulemay also up-cast each record as a first type of record and send eachup-casted record to the first node, which is still executing the firstcontinuous query. Thus, the first node may continue processing a firsttype of records and the new, second type of records without beingsuspended and/or reconfigured. Furthermore, each second type of recordmay be processed concurrently by the first node and the second node.

4.0 Implementation Mechanisms—Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 4 is a block diagram that illustrates a computersystem 400 upon which an embodiment of the disclosure may beimplemented. Computer system 400 includes a bus 402 or othercommunication mechanism for communicating information, and a hardwareprocessor 404 coupled with bus 402 for processing information. Hardwareprocessor 404 may be, for example, a general purpose microprocessor.

Computer system 400 also includes a main memory 406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 402for storing information and instructions to be executed by processor404. Main memory 406 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 404. Such instructions, when stored innon-transitory storage media accessible to processor 404, rendercomputer system 400 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 400 further includes a read only memory (ROM) 408 orother static storage device coupled to bus 402 for storing staticinformation and instructions for processor 404. A storage device 410,such as a magnetic disk or optical disk, is provided and coupled to bus402 for storing information and instructions.

Computer system 400 may be coupled via bus 402 to a display 412, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 414, including alphanumeric and other keys, is coupledto bus 402 for communicating information and command selections toprocessor 404. Another type of user input device is cursor control 416,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 404 and forcontrolling cursor movement on display 412. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 400 in response to processor 404 executing one or more sequencesof one or more instructions contained in main memory 406. Suchinstructions may be read into main memory 406 from another storagemedium, such as storage device 410. Execution of the sequences ofinstructions contained in main memory 406 causes processor 404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 410.Volatile media includes dynamic memory, such as main memory 406. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 404 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 402. Bus 402 carries the data tomain memory 406, from which processor 404 retrieves and executes theinstructions. The instructions received by main memory 406 mayoptionally be stored on storage device 410 either before or afterexecution by processor 404.

Computer system 400 also includes a communication interface 418 coupledto bus 402. Communication interface 418 provides a two-way datacommunication coupling to a network link 420 that is connected to alocal network 422. For example, communication interface 418 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 418 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 418sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424 orto data equipment operated by an Internet Service Provider (ISP) 426.ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 428. Local network 422 and Internet 428 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, includingprogram code, through the network(s), network link 420 and communicationinterface 418. In the Internet example, a server 430 might transmit arequested code for an application program through Internet 428, ISP 426,local network 422 and communication interface 418.

The received code may be executed by processor 404 as it is received,and/or stored in storage device 410, or other non-volatile storage forlater execution.

5.0 Extensions and Alternatives

In the foregoing specification, embodiments of the disclosure have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the disclosure, and is intended by the applicants to be thedisclosure, is the set of claims that issue from this application, inthe specific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

What is claimed is:
 1. A computer system comprising: a memory; aprocessor coupled to the memory; processor logic coupled to theprocessor and the memory configured to: receive a first recordcomprising one or more first fields from a first source computer; inresponse to determining that the first record is of a first type, sendthe first record to a first node computer that processes records of thefirst type but is suspended when processing records of a second type;receiving, from the first source computer, an updated first recordcomprising the first record and one or more new fields from the firstsource computer; in response to determining that the updated firstrecord is of the second type, sending the updated first record to asecond node computer that processes records of the second type; aftersending the first record to the first node computer and the updatedfirst record to the second node computer, determining whether the secondtype is a first subtype of the first type; in response to determiningthat the second type is the first subtype of the first type, send theupdated first record to the first node computer to be processed, withoutsuspending and reconfiguring the first node computer.
 2. The computersystem of claim 1, wherein the processor logic is configured to: inresponse to determining that the first record is the first type and thefirst node computer is associated with the first type, send a firstsignature with the first record to the first node computer; in responseto determining that the updated first record is the second type and thesecond node computer is associated with the second type, send a secondsignature with the updated first record to the first node computer; inresponse to determining that the second type is the first subtype of thefirst type, send the first signature with the updated first record tothe first node computer.
 3. The computer system of claim 1, wherein theprocessor logic is configured to: organize the one or more first fieldsin the first record according to one or more first indices; organize theone or more first fields in the updated first record according to theone or more first indices; organize the one or more new fields in theupdated first record according to one or more second indices, whereineach index in the one or more first indices references a memory addressthat is before the memory address referenced by each index in the one ormore second indices.
 4. The computer system of claim 1 comprising athird node computer comprising a third processor and a third memoryconfigured to process one or more third type of records based on a thirdoperation, wherein the processor logic is configured to: receive, from athird source computer, a third record comprising the one or more firstfields, the one or more new fields, and one or more third fields; inresponse to determining that the third record is a third type and thethird node computer is associated with the third type, send the thirdrecord to the third node computer to be processed; in response todetermining that the third type is a second subtype of the second type,send the third record to the second node computer to be processedwithout suspending and reconfiguring the second node computer; inresponse to determining that the second type is the first subtype of thefirst type, send the third record to the first node computer to beprocessed without suspending and reconfiguring the second node computer.5. The computer system of claim 1, wherein the processor logic isconfigured to: determine whether the first record is the first typebased on the one or more first fields included in the first record;determine whether the updated first record is the second type based onthe one or more first fields and the one or more new fields included inthe updated first record; determine whether the second type is the firstsubtype of the first type based on each field of the one or more firstfields being included in the updated first record.
 6. The computersystem of claim 1, wherein the processor logic is configured to:associate a first signature with the first type; associate a secondsignature with the second type; associate the second signature with thefirst signature as the first subtype; determine the first record has thefirst signature based on one or more first field names of the one ormore first fields, which are included in the first record; determinewhether the first record is the first type based on the first signature;determine the updated first record has the second signature based on oneor more second field names of the one or more first fields and the oneor more new fields, which are included in the updated first record;determine whether the updated first record is the second type based onthe second signature; determine whether the second type is the firstsubtype of the first type based on the second signature being associatedwith the first signature; in response to determining that the secondtype is the first subtype of the first type, send the first signaturewith the updated first record to the first node computer.
 7. Thecomputer system of claim 1, wherein the processor logic is configuredto: associate a first signature with the first type; associate a secondsignature with the second type; associate the second signature with thefirst signature as the first subtype; determine the first record has thefirst signature based on one or more first data structures in the one ormore first fields, which are included in the first record; determinewhether the first record is the first type based on the first signature;determine the updated first record has the second signature based on oneor more second data structures in the one or more first fields and theone or more new fields, which are included in the updated first record;determine whether the updated first record is the second type based onthe second signature; determine whether the second type is the firstsubtype of the first type based on the second signature being associatedwith the first signature; in response to determining that the secondtype is the first subtype of the first type, send the first signaturewith the updated first record to the first node computer.
 8. Thecomputer system of claim 1, wherein the processor logic is configuredto: determine whether the first record is the first type based onwhether the first record was received from a first source; determinewhether the updated first record is the second type based on whether thesecond record was received from a second source.
 9. The computer systemof claim 1, wherein: the first node computer comprising a firstprocessor and a first memory configured to process one or more firsttype of records based on a first operation; the second node computercomprising a second processor and a second memory configured to processone or more second type of records based on a second operation; theprocessor logic is configured to: determine the second node computer hasfinished processing the updated first record; send the updated firstrecord to the first node computer after determining the second nodecomputer has finished processing the updated first record.
 10. Thecomputer system of claim 9, wherein the processor logic is configuredto: send the updated first record to one or more sub-node computersassociated with the second node computer in a particular order;determine that each sub-node computer of the one or more sub-nodecomputers has processed the updated first record in the particularorder; determine the second node computer is finished processing theupdated first record comprises after determining that each sub-nodecomputer of the one or more sub-node computers has processed the updatedfirst record in the particular order.
 11. A method comprising: receivinga first record comprising one or more first fields from a first sourcecomputer; in response to determining that the first record is of a firsttype, send the first record to a first node computer that processesrecords of the first type but is suspended when processing records of asecond type; receiving, from the first source computer, an updated firstrecord comprising the first record and one or more new fields from thefirst source computer; in response to determining that the updated firstrecord is of the second type, sending the updated first record to asecond node computer that processes records of the second type; aftersending the first record to the first node computer and the updatedfirst record to the second node computer, determining whether the secondtype is a first subtype of the first type; in response to determiningthat the second type is the first subtype of the first type, send theupdated first record to the first node computer to be processed, withoutsuspending and reconfiguring the first node computer. wherein the methodis performed by one or more computing devices.
 12. The method of claim11 comprising: in response to determining that the first record is thefirst type and the first node computer is associated with the firsttype, send a first signature with the first record to the first nodecomputer; in response to determining that the updated first record isthe second type and the second node computer is associated with thesecond type, send a second signature with the updated first record tothe first node computer; in response to determining that the second typeis the first subtype of the first type, send the first signature withthe updated first record to the first node computer.
 13. The method ofclaim 11 comprising: organizing the one or more first fields in thefirst record according to one or more first indices; organizing the oneor more first fields in the updated first record according to the one ormore first indices; organizing the one or more new fields in the updatedfirst record according to one or more second indices, wherein each indexin the one or more first indices references a memory address that isbefore the memory address referenced by each index in the one or moresecond indices.
 14. The method of claim 11 comprising: receiving, from athird source computer, a third record comprising the one or more firstfields, the one or more new fields, and one or more third fields; inresponse to determining that the third record is a third type and athird node computer is associated with the third type, send the thirdrecord to the third node computer to be processed; in response todetermining that the third type is a second subtype of the second type,send the third record to the second node computer to be processedwithout suspending and reconfiguring the second node computer; inresponse to determining that the second type is the first subtype of thefirst type, send the third record to the first node computer to beprocessed without suspending and reconfiguring the second node computer.15. The method of claim 11 comprising: determining whether the firstrecord is the first type based on the one or more first fields includedin the first record; determining whether the updated first record is thesecond type based on the one or more first fields and the one or morenew fields included in the updated first record; determining whether thesecond type is the first subtype of the first type based on each fieldof the one or more first fields being included in the updated firstrecord.
 16. The method of claim 11 comprising: associating a firstsignature with the first type; associating a second signature with thesecond type; associating the second signature with the first signatureas the first subtype; determining the first record has the firstsignature based on one or more first field names of the one or morefirst fields, which are included in the first record; determiningwhether the first record is the first type based on the first signature;determining the updated first record has the second signature based onone or more second field names of the one or more first fields and theone or more new fields, which are included in the updated first record;determining whether the updated first record is the second type based onthe second signature; determining whether the second type is the firstsubtype of the first type based on the second signature being associatedwith the first signature; in response to determining that the secondtype is the first subtype of the first type, send the first signaturewith the updated first record to the first node computer.
 17. The methodof claim 11 comprising: associating a first signature with the firsttype; associating a second signature with the second type; associatingthe second signature with the first signature as the first subtype;determining the first record has the first signature based on one ormore first data structures in the one or more first fields, which areincluded in the first record; determining whether the first record isthe first type based on the first signature; determining the updatedfirst record has the second signature based on one or more second datastructures in the one or more first fields and the one or more newfields, which are included in the updated first record; determiningwhether the updated first record is the second type based on the secondsignature; determining whether the second type is the first subtype ofthe first type based on the second signature being associated with thefirst signature; in response to determining that the second type is thefirst subtype of the first type, send the first signature with theupdated first record to the first node computer.
 18. The method of claim11 comprising: determining whether the first record is the first typebased on whether the first record was received from a first source;determining whether the updated first record is the second type based onwhether the second record was received from a second source.
 19. Themethod of claim 11 comprising: determining the second node computer hasfinished processing the updated first record; wherein sending theupdated first record to the first node computer after determining thesecond node computer has finished processing the updated first record.20. The method of claim 19 comprising: sending the updated first recordto one or more sub-node computers associated with the second nodecomputer in a particular order; determining that each sub-node computerof the one or more sub-node computers has processed the updated firstrecord in the particular order; determining the second node computer isfinished processing the updated first record comprises after determiningthat each sub-node computer of the one or more sub-node computers hasprocessed the updated first record in the particular order.